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1  Introduction  and  Overview 

1.1  Introduction 

In  Joint  Vision  2010  [1],  the  Chairman  of  the  Joint  Chiefs  of  Staff  outlined  a  vision  of  effective, 
efficient  armed  services  for  the  next  century.  This  document  stressed  the  importance  of  “infor¬ 
mation  superiority”  which  he  defined  as  “The  capability  to  collect,  process,  and  disseminate  an 
uninterrupted  flow  of  information  while  exploiting  or  denying  and  adversary’s  ability  to  do  the 
same.”  Enhanced  command,  control,  and  intelligence  provided  by  information  superiority  will 
transform  the  traditional  functions  of  maneuver,  strike,  protection  and  logistics  into  a  new  concep¬ 
tual  framework  for  operations.  The  document  identified  the  tactical  use  of  autonomous  assets  as 
a  key  technology  that  will  enable  the  commanders  to  conduct  dominant  maneuvers  and  precision 
engagements. 

Since  the  issuance  of  [1],  the  United  States  has  been  involved  in  military  conflicts  in  Afghanistan 
and  Iraq,  that  saw  increased  use  of  autonomous  tactical  aircraft  such  as  Predators  and  Global 
Hawks  for  surveillance  and  target  prosecution.  In  addition,  the  U.  S.  Air  Force  has  proceeded  with 
the  development  of  Unmanned  Combat  Air  Vehicles  (UAVs)  in  order  to  accomplish  missions  that 
may  be  difficult  for  manned  aircraft  due  to  survivability  or  other  reasons.  These  missions  include 
Suppression  of  Enemy  Air  Defenses  (SEAD),  moving  target  attack,  fixed  target  attack.  Intelligence 
Surveillance  and  Reconnaissance  (ISR),  jamming,  theater  missile  defense,  and  counter  weapons  of 
mass  destruction.  UAV  prototypes  have  demonstrated  so  far  the  capability  for  unmanned  flight. 
Other  services  such  as  the  U.  S.  Army  and  U.  S.  Navy  have  been  actively  pursuing  the  development 
of  unmanned  combat  air,  ground  and  underwater  vehicles. 

One  of  the  major  limitations  of  current  tactical  unmanned  air  vehicles  (UAVs)  is  the  high  level  of 
human  control  required  to  operate  a  single  vehicle.  Currently,  aircraft  such  as  Predators  require  two 
or  more  operators  per  aircraft  to  provide  information  processing,  navigation  and  mission  control 
decisions;  this  places  limits  on  the  use  of  large  numbers  of  autonomous  assets  acting  in  a  coordinated 
manner;  the  use  of  unmanned  air  vehicles  in  Afghanistan  and  Iraq  consisted  primarily  of  individual 
missions  involving  single  aircraft.  The  new  range  of  UAVs  will  have  the  capability  of  autonomous 
sensing  and  control,  and  will  be  capable  of  cooperatively  detecting,  locating,  classifying  and 
prosecuting  intelligent,  mobile  targets. 
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In  order  to  achieve  the  full  potential  of  unmanned  tactical  vehicles,  many  of  the  decisions  made  by 
human  operators  will  be  generated  by  an  intelligent  cooperative  control  system,  guided  by  objectives 
and  constraints  provided  by  human  operators.  This  cooperative  control  will  have  to  solve  complex 
dynamic  decision  problems  associated  with  mission  planning  and  control,  in  an  unstructured  and 
uncertain  environment,  in  near  real  time,  and  without  human  intervention  [17,  18,  19],  adapting  in 
an  intelligent  fashion  to  events  which  arise  in  a  hostile,  uncertain  and  rapidly  evolving  environment. 
This  requires  new  technology  for  cooperative  mission  control^  capable  of  selecting  and  coordinating 
the  activities  of  multiple  heterogeneous  platforms  to  achieve  a  common  objective. 

This  report  documents  the  results  of  our  investigations  towards  a  theory  of  cooperative  mission 
control  for  groups  of  unmanned  air  vehicles.  The  investigations  were  focused  on  two  levels  of 
control:  (1)  Strategy  Development,  to  determine  missions  of  interest  and  corresponding  strategies 
for  accomplishing  those  missions  in  centralized  and  distributed  settings,  and  (2)  Dynamic  Vehicle 
Assignment  and  Scheduling,  which  determines  activities,  routes  and  schedules  for  individual  vehicles 
to  accomplish  desired  missions  of  interest  and  strategies.  The  main  results  of  our  research  were: 

•  A  new  theory  for  developing  strategies  for  target  prosecution  among  heterogeneous  vehicles 
that  use  unreliable  resources  [5,  10].  The  results  are  based  on  a  stochastic  dynamic  program¬ 
ming  approach,  and  use  a  constraint  relaxation  approach  to  generate  approximate  solutions 
that  are  made  feasible  through  a  model-predictive  control  approach.  These  results  were  ex¬ 
tended  to  include  dynamic  sensor  management  near  the  end  of  the  program. 

•  New  asynchronous  distributed  algorithms  for  modifying  task  assignments  among  members  of 
UAV  teams  in  response  to  dynamic  scenarios  where  tasks  arrive  and  depart,  when  information 
among  team  members  can  differ  due  to  communication  delays  [9].  Our  results  in  [9]  estab¬ 
lish  that  the  asynchronous  algorithms  have  proven  stability  properties,  such  as  guaranteed 
convergence  to  optimal  assignments  once  scenarios  stabilize. 

•  A  receding  horizon  control  for  simultaneous  path  planning  and  task  assignment  [6,  8]  that  is 
related  to  potential  function  approaches  to  path  planning  [7].  Assignment  occurs  implicitly 
as  vehicles  are  attracted  to  tasks,  and  avoid  overlap  with  other  vehicles.  The  inputs  to  this 
algorithm  axe  the  specific  tasks  associated  with  the  strategies  selected  in  [10]. 

The  above  results  were  provided  to  industry  researchers  for  use  in  experimental  cooperative  control 
programs  such  as  DARPA’s  MICA  and  AFRL  DACC  and  DCAT  programs.  These  collaborations 


2 


enabled  us  to  evaluate  the  relevance  of  the  results  without  incurring  the  commensurate  cost  in 
developing  and  using  costly  simulations.  These  industry  researchers  are  actively  involved  in  devel¬ 
oping  mission  control  software  for  programs  such  as  J-UCAS,  and  provide  a  transition  path  for  our 
results. 

The  results  reported  below  include  the  works  of  the  two  principal  investigators,  Prof.  D.  Castahon 
and  Prof.  C.  Cassandras  and  their  three  Ph.  D.  students,  Mr.  Wei  Li,  Mr.  Xiang  Ma  and  Major 
Darryl  Ahner,  US  Army.  The  three  students  are  scheduled  to  complete  their  dissertations  in  2005. 
The  following  sections  provide  an  overview  of  the  key  results  of  the  research. 
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2  Strategy  Development 


2.1  Formulation 

An  important  focus  of  the  research  was  dynamic  of  UAV  activities  in  the  presence  of  uncertainty. 
In  particular,  we  were  interested  in  models  where  the  probability  distribution  of  future  uncertain 
events  depends  explicitly  on  the  current  choice  of  actions.  Such  models  capture  effects  such  as 
the  risk  incurred  by  UAVs  in  selecting  alternate  paths,  or  the  uncertain  effects  of  actions  such  as 
attacking  targets  where  the  outcome  is  uncertain. 

Our  initial  investigations  focused  on  abstractions  of  dynamic  scheduling  problems  for  a  group  of 
autonomous  vehicles  carrying  nonrenewable  resources  (e.g.  munitions)  that  must  be  assigned  to 
tasks  which  arrive  in  a  dynamic,  uncertain  fashion.  Furthermore,  resources  may  fail  to  successfully 
complete  a  task.  The  mission  control  problem  in  this  context  is  to  schedule  resources  over  time  to 
assign  to  individual  tasks  based  on  vehicle  capabilities,  available  resources  and  available  information, 
so  as  to  maximize  the  expected  task  completion  value.  We  assume  in  this  framework  that  vehicles 
are  able  to  observe  resource  failures  during  each  time  period  through  mechanisms  such  as  battle 
damage  assessment.  Thus,  one  can  exploit  this  dynamic  information  and  adapt  future  decisions 
based  on  the  observation  of  success  or  failure  events. 

The  problem  we  focused  on  was  a  stochastic  control  problem  with  multiple  stages.  The  simplest 
two-stage  version  of  the  problem  is  summarized  as  follows:  Assume  that  there  are  N  tasks,  indexed 
by  i  =  1, . . . ,  A/',  and  that  there  are  a  total  of  M  non-renewable  homogeneous  resources  which  can  be 
assigned  to  each  task  over  two  possible  stages.  Each  task  has  a  numerical  value  Vi  which  is  obtained 
by  completing  the  task.  Each  resource  has  a  cost  C  of  assigning  the  resource.  When  a  resource 
is  assigned  to  task  i  in  stage  A;,  the  event  that  the  resource  completes  the  task  successfully  has 
probability  Pi(fc),  and  this  event  is  independent  of  events  generated  by  other  resource  assignments. 

Let  Xi{l)  denote  the  number  of  resources  assigned  to  task  i  at  stage  1.  The  probability  that  task  i  is 
not  completed  by  these  resource  assignments  is  obtained  from  the  above  independence  assumptions 
as: 

P5(i,l)  =  (l-Pi(l)r('=)  (1) 

At  the  end  of  stage  1,  the  set  of  completed  tasks  will  be  observed.  This  set  will  denote  the  task 
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state  vector,  and  is  a  Boolean  vector  ^  G  =  {0, 1}^;  uJi  =  0  denotes  that  task  i  was  completed 
in  stage  1,  and  ui  —  1  denotes  the  complementary  event  for  task  i.  Given  a  vector  of  stage  1 
resource  allocations  x(l),  equation  (1)  induces  a  probability  distribution  P(^|x(l))  on  the  possible 
outcomes.  The  stage  2  allocations  are  strategies  which  depend  on  the  specific  outcome  a;  which  is 
observed ,  as  X  ( 2 ,  . 

The  evolution  of  the  task  state  w  across  stages  is  a  Markov  process  with  states  in  f).  Given  resource 
allocations  x(l)  and  recourse  strategies  ^(2,^),  the  probability  that  task  i  is  not  completed  either 
in  stage  1  or  in  stage  2  is  given  by 

Ps{i,  2)  =  Pico\x{l))I{uji  =  1)(1  -  (2) 

where  /(•)  is  the  indicator  function,  and 

pyx(i))=  n  n  (3) 

The  optimal  control  problem  is  to  select  resource  allocations  x(l)  and  recourse  strategies  x{2^u) 
to  minimize  the  expected  incomplete  task  value  plus  the  expected  cost  of  using  resources: 

N  N 

E{J2ViPsii,2)+CY,{xiil)  +  Xi{2,^)}  (4) 

i=l  i=l 

subject  to  the  constraints 

N 

^(xi(l)  +  Xi{2^cj))  <  M  for  all  ^  G  H  .(5) 

i=i 

Xi{l)  G  {0, 1, . . .  ^M}yXi{2^u))  G  {0, 1, . . .  ,M}  for  all  cj  G  fi, 2  G  {1, . . .  ,n}  (6) 

The  above  problem  is  a  two-stage  stochastic  control  problem  with  a  discrete  state  space  that  grows 
exponentially  in  the  number  of  tasks  and  a  discrete  decision  space  which  also  grows  exponentially 
in  the  number  of  tasks. 

2.2  Model  Predictive  Control  Algorithms 

In  our  papers  [5,  10],  we  developed  the  stochastic  dynamic  programming  solution  to  the  above 
problem.  This  solution  is  impractical  for  real-time  mission  control  algorithms.  As  an  alternative, 
we  developed  a  new  formulation  based  on  techniques  from  model  predictive  control  (MFC).  Instead 
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of  using  the  model  in  (l)-(6),  we  solve  a  relaxed  version  of  the  problem  obtained  by  replacing  the 
2-^  constraints  in  eq.(5)  by  one  average  resource  utilization  constraint: 

N 

+  ^  -W  (7) 

i— 1 

The  new  optimization  problem  used  for  the  MFC  approach  is  to  minimize  (4)  subject  to  constraints 
in  (7,6).  The  solution  of  this  optimization  problem  determines  x(l),  the  first  stage  resource  alloca¬ 
tions,  as  well  as  strategies  for  future  allocations.  Only  the  first  stage  allocations  are  implemented; 
subsequently,  based  on  the  observed  outcome  w,  the  second  stage  allocations  are  determined  using 
a  static  network  optimization  problem. 

Note  that  every  feasible  strategy  for  the  constraints  in  (5)  also  satisfies  (7).  Thus,  the  MFC  problem 
is  a  relaxation  of  the  original  problem,  expanding  the  set  of  admissible  strategies,  and  its  solution 
is  an  optimistic  lower  bound  on  the  optimal  performance  in  the  original  problem. 

The  main  theoretical  result  in  [5,  10]  is  an  equivalence  result  that  establishes  the  existence  of 
optimal  strategies  that  use  only  “local”  feedback  information.  That  is,  the  resource  allocations 
in  stage  2  for  task  i,  denoted  as  Xi{2^u))^  can  be  restricted  to  depend  only  on  the  state  of  task 
uji.  This  allows  the  use  of  Lagrange  duality  to  achieve  a  separable  solution  among  single  task 
problems,  coordinated  through  dual  prices.  The  result  is  an  0{N  log  N)  algorithm  for  exact  solution 
of  the  two-stage  relaxed  integer  stochastic  dynamic  programming  problem,  for  N  tasks.  This  fast 
algorithm  yields  approximate  first  stage  decisions  in  the  MFC  approach.  Once  the  outcome  of 
the  first  stage  actions  are  observed,  the  second  stage  decisions  are  re-optimized  using  another  fast 
algorithm. 

2.3  Numerical  Experiments 

In  order  to  evaluate  the  effectiveness  of  the  proposed  MFC  algorithms,  we  conducted  several  ex¬ 
periments  comparing  the  following  algorithms: 

1.  The  exact  stochastic  dynamic  programming  (SDF)  solution,  obtained  by  enumerating  the 
possible  first  stage  allocations  and  finding  a  global  minimum. 

2.  An  approximate  exponential  complexity  algorithm  (lA)  described  in  [5,  10]. 
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Tasks 

Resources 

lA  Alg. 

MPC  Alg. 

Worst  MPC 

7 

7 

100% 

99.92% 

98.6% 

7 

9 

100% 

99.82% 

99.18% 

7 

11 

100% 

99.996% 

99.86% 

9 

7 

98.30% 

9 

9 

99.48% 

9 

11 

11 

7 

HOB 

99.53% 

11 

9 

11 

11 

100% 

99.74% 

99.19% 

Table  1:  Performance  of  MPC  and  lA  algorithms  as  percent  of  value  completed  by  SDP 


3.  The  MPC  algorithm  developed  in  [5,  10]  discussed  above. 

The  first  set  of  experiments  consisted  of  random  problems  with  7  to  11  tasks,  with  task  values 
selected  randomly  in  the  range  of  1-10,  and  task  success  probabilities  selected  randomly  in  the 
interval  [0.7, 0.9].  The  number  of  resources  varied  from  7  to  11.  For  each  experiment,  we  generated 
100  random  problems,  and  obtained  the  solutions  (in  terms  of  value  achieved)  given  by  the  SDP, 
lA  and  MPC  algorithms.  We  computed  the  performance  of  the  two  suboptimal  algorithms  as 
percentage  of  the  value  achieved  by  the  optimal  SDP  algorithm,  averaged  over  the  100  problems. 
We  also  computed  the  worst  case  percentage  difference  in  performance  between  the  MPC  algorithm 
and  the  SDP  algorithm.  The  results  are  summarized  in  Table  1. 

The  results  in  Table  I  establish  that  the  MPC  algorithm  yields  near-optimal  performance:  The  worst 
case  performance  across  900  problems  tested  was  within  2%  of  the  optimal  SDP  performance,  and 
the  average  performance  was  within  0.3%  of  the  optimal  SDP  performance. 

The  second  set  of  experiments  used  problems  with  16  and  20  tasks,  and  with  a  varying  number  of 
resources  from  12  to  20.  For  these  problems,  computing  the  exact  dynamic  programming  solution 
using  enumerative  techniques  was  prohibitively  long.  As  a  reference  point,  it  required  3  days  on  a 
LINUX  Pentium  1.7  GHz  workstation  to  solve  100  instances  of  the  11  task  problem.  We  compared 
results  only  for  lA  and  MPC  algorithms.  The  statistics  reported  are  the  percentage  of  the  lA 
value  that  is  achieved  by  the  MPC  algorithm.  We  report  both  the  average  percentage  across  100 
problems  and  the  worst-case  percentage  over  the  100  problems  for  each  experiment.  The  results 
are  summarized  in  Table  11. 
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Tasks 

Resources 

Ave.  MPC 

Worst  MPC 

16 

12 

99.81% 

99.22% 

16 

99.82% 

99.33% 

■■ 

20 

12 

99.85% 

16 

99.85% 

20 

20 

99.88% 

99.37% 

Table  2:  Performance  of  MPC  algorithm  as  percent  of  value  completed  by  lA  algorithm 


The  results  in  Table  II  confirm  the  near  optimal  behavior  of  the  Model  Predictive  Control  algorithm. 
The  average  performance  is  within  0.2%  of  the  performance  of  the  lA  algorithm,  and  the  worst 
case  performance  is  within  1%  of  the  performance  of  the  I A  algorithm.  In  terms  of  computation 
time,  the  I A  algorithm  requires  over  13  minutes  to  solve  a  single  instance  of  a  20  task,  20  resource 
problem  on  a  Pentium  1.4  GHz  workstation  running  Linux.  In  contrast,  the  MPC  algorithm  solved 
100  instances  of  1000  task,  1000  resource  problems  in  a  total  of  3,5  seconds.  This  is  nearly  five 
orders  of  magnitude  faster,  making  it  suitable  for  real-time  mission  control. 

2.4  Extensions 

The  above  results  have  been  extended  in  several  significant  directions,  as  discussed  in  [5,  10].  The 
first  extension  is  inclusion'  of  multiple  period  dynamics,  extending  the  two-period  results  above. 
The  second  extension  is  to  allow  for  new  target  arrivals  in  future  periods.  A  third  extension  is  to 
consider  targets  with  unknown  values,  that  require  the  use  of  sensors  to  confirm  their  true  value. 
This  extension  corresponds  to  targets  with  uncertain  identity,  that  must  be  identified  in  order 
to  determine  their  true  value.  A  fourth  extension  was  to  have  different  classes  of  non-renewable 
resources,  corresponding  to  different  types  of  munitions. 

For  each  of  the  above  extensions,  we  developed  fast  MPC  algorithms,  and  evaluated  the  performance 
of  these  algorithms  on  simulated  problems.  An  experiment  documented  in  [5,  10]  used  two  different 
types  of  munitions.  Each  type  j  had  different  effectiveness  pij  for  each  task  i.  Task  values  were 
generated  randomly  in  the  range  of  1  to  10,  with  random  values  for  pij  in  the  range  of  0.7  to 
0.9.  We  compared  the  MPC  performance  after  averaging  over  10,000  Monte  Carlo  trials  to  the 
optimistic  bound  obtained  by  the  solution  of  the  approximate  model  used  by  the  MPC  controller, 
for  100  random  instances  each  of  10  task,  50  task  and  100  task  problems.  The  number  of  available 
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Tasks 

Res.  I 

Res.  II 

Ave.  MPC 

Worst  MPC 

10 

5 

5 

98.5% 

95.8% 

25 

25 

99.5% 

99.2% 

50 

50 

99.7% 

97.9% 

Table  3:  Performance  of  MPC  algorithm  as  percentage  of  task  value  completed  by  Mixed  Strategy 
upper  bound  for  two  resource  types 

resources  in  each  problem  was  set  so  that  the  total  number  of  resources  equaled  the  number  of 
tasks,  and  there  were  equal  numbers  of  two  types  of  resources. 

The  results  of  this  experiment,  summarized  in  Table  III,  show  that  the  MPC  controller  achieves 
expected  performance  is  over  95%  of  an  optimistic  upper  bound  to  the  optimal  performance  in 
every  problem  instance  tested.  On  average,  the  expected  performance  is  98.5%  or  higher  for  the 
different  problem  sizes.  Note  that  the  average  performance  improves  with  increasing  problem  size, 
as  statistical  mixing  across  tasks  increases  the  accuracy  of  the  approximate  optimization  problem 
used  by  the  MPC  algorithm. 

2.5  Continuing  Directions  of  Investigation 

The  majority  of  our  effort  was  focused  on  the  allocation  of  nonrenewable  resources  such  as  muni¬ 
tions.  Our  results  provide  a  design  theory  for  real-time  mission  control  algorithms  that  determine 
sequencing  of  targeting  activities  over  temporal  periods,  taking  into  account  uncertainty  in  success 
of  munitions.  However,  the  above  models  do  not  address  the  use  of  UAVs  for  surveillance  activities, 
where  the  expended  resources  are  renewable,  such  as  sensing  resources. 

In  the  last  year  of  our  research  effort,  we  studied  a  different  class  of  models:  Strategy  Development 
for  sensor  assignment,  based  on  generalization  of  surveillance  problems  involving  tactical  UAVs 
such  as  Predators  or  smaller  UAVs.  In  these  problems,  UAV  sensors  search  for  an  unknown  set 
of  mobile,  non-cooperative  targets,  and  attempt  to  detect,  track  and  classify  these  targets.  Our 
focus  was  on  problems  where  tasks  arrive  dynamically  using  stochastic  models,  and  where  there  is 
significant  risk  incurred  by  the  individual  vehicles.  We  developed  a  mathematical  representation  of 
these  problems  that  capture  elements  of  both  risk  to  UAVs,  and  uncertainty  in  future  arrivals  of 
tasks.  We  developed  control  algorithms  for  such  problems  based  on  receding  horizon  planning  using 
a  mixed  integer  linear  programming  formulation,  as  well  as  alternative  approaches  using  Lagrangian 
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relaxation  and  rollout  algorithms  [20].  Initial  results  were  presented  in  [llj.  The  results  of  this 
research  form  the  core  of  Major  Darryl  Ahner’s  doctoral  dissertation  at  Boston  University,  which 
will  be  completed  May  2005, 

2.6  Distributed  Algorithms 

The  previous  discussion  focused  on  mission  control  algorithms  that  would  be  executed  at  a  central 
location,  such  as  a  ground-control  station,  and  thus  assume  centralization  of  information.  In  order 
to  address  issues  of  distributed  mission  control,  where  multiple  agents  have  access  to  different  infor¬ 
mation,  we  studied  a  class  of  distributed  task  assignment  problems  where  multiple  UAVs  cooperate 
to  solve  optimization  problems  using  only  local,  possibly  outdated  information.  Specifically,  we 
studied  the  problem  of  task  partitioning  among  members  of  a  team  of  cooperating  agents,  each 
with  potentially  different  local  information.  In  these  problems,  each  agent  has  finite  resources, 
and  has  an  initial  set  of  tasks  assigned  to  it  that  represents  an  optimal  partitioning  given  the 
initial  centrally  known  information.  The  problem  of  interest  is  stated  as  follows:  Given  a  finite 
sequence  of  new  task  arrivals,  observed  locally  by  different  agents  at  different  times,  find  an  optimal 
redistribution  of  the  old  and  new  tasks  among  the  agents  in  the  presence  of  different  information. 

One  possible  approach  to  this  problem  is  to  wait  until  all  of  the  local  information  is  centrally 
acknowledged  by  all  agents.  At  this  time,  every  agent  could  execute  an  optimal  reassignment 
algorithm  to  compute  the  new  task  assignments.  However,  this  approach  can  incur  significant  delays 
waiting  to  obtain  synchronization  of  information.  An  alternative  is  to  allow  each  agent  to  compute 
modifications  to  the  current  ask  assignments  without  waiting  for  information  synchronization.  Each 
agent  computes  a  speculative  adjustment  to  assignments  in  response  to  each  new  task  it  detects, 
and  communicates  these  adjustments  to  other  team  members. 

In  [9],  we  explored  this  approach,  and  developed  a  distributed,  asynchronous  reassignment  algorithm 
that  responds  to  new  events  locally  without  waiting  for  information  synchronization.  We  extended 
previous  results  in  asynchronous  algorithms  for  parallel  solution  of  assignment  problems  [3,  4,  2] 
that  addressed  problems  with  known  tasks  to  the  case  where  tasks  are  dynamically  arriving  and 
information  differences  occur  among  agents.  Specifically,  we  developed  a  new  class  of  asynchronous 
task  algorithms  that  generalize  the  distributed  sequential  shortest  path  algorithms  of  [4,  2].  In 
our  algorithms,  agents  respond  to  the  discovery  of  new  tasks  by  suggesting  redistribution  of  tasks 
among  agents.  Since  agents  observe  arrivals  of  the  same  task  at  different  times,  our  algorithm  in 
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[9]  often  encounters  multiple  agents  reacting  to  the  same  task,  and  includes  a  protocol  for  conflict 
resolution  as  well  ^  information  communication  among  agents. 

Our  main  result  shows  that,  as  long  as  communication  delays  are  bounded,  the  distributed  assign¬ 
ment  algorithms  converge  to  an  optimal  assignment  once  all  the  tasks  have  arrived.  Furthermore, 
the  distributed  algorithms  are  constantly  shifting  responsibility  for  tasks  locally  between  agents  in 
response  to  new  task  arrivals  that  are  detected  only  locally,  instead  of  waiting  for  all  information 
to  be  distributed  globally.  The  convergence  results  depend  critically  on  the  presence  of  a  validation 
agent,  whose  role  it  is  to  reject  assignment  modifications  suggested  by  agents  when  the  information 
used  to  generate  such  modifications  is  deemed  to  be  too  far  out-of-date.  Our  algorithms  describe  a 
validation  protocol  that  guarantees  that  a  complementary  slackness  condition  is  maintained  by  all 
accepted  changes.  This  invariance  is  exploited  to  prove  the  convergence  to  optimal  reassignments 
once  new  task  arrivals  stop. 

The  distributed  algorithms  have  a  significant  advantage  over  centralized  algorithms:  They  respond 
promptly  to  detection  of  new  events,  without  delays  required  to  synchronize  information  with  other 
agents.  To  illustrate  the  performance  of  the  distributed  algorithm,  we  ran  a  simple  experiment  with 
10  agents,  each  carrying  10  resources.  The  vehicles  were  evenly  distributed  in  a  square  grid.  There 
were  up  to  100  task  arrivals,  randomly  distributed  with  a  uniform  distribution  on  the  square  grid. 
Each  task  arrival  was  observed  only  by  the  closest  agent  to  it.  The  value  of  each  task  assignment  to 
each  agent  was  inversely  proportional  to  the  distance  from  each  agent,  plus  a  random  component 
with  standard  deviation  equal  to  10%  of  the  maximum  value. 

When  an  agent  detected  a  task  arrival,  it  computed  desired  modifications  to  task  assignments 
and  waited  for  a  turn  to  communicate  with  the  validation  agent.  We  simulated  a  cyclic  slotted 
communication  system,  where  each  agent  can  communicate  with  the  validation  agent  once  a  cycle, 
which  occurs  after  a  fixed  number  of  external  task  arrivals.  The  agents  communicate  in  fixed  order; 
agents  that  communicate  first  are  unaware  of  information  provided  by  agents  that  communicate 
later.  This  creates  asymmetries  in  information  due  to  two  effects:  between  communication  slots, 
new  task  arrivals  are  known  only  to  their  closest  agents,  and  during  communication  times,  agents 
only  know  the  information  provided  to  the  validation  agent  by  other  agents  that  communicated 
earlier  in  the  cycle. 

Table  4  shows  the  averaged  results  of  Monte  Carlo  experiments  respect  to  task  arrivals.  The  results 
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Total  Number  of  Tasks 

Arrivals/cycle 

75 

80 

85 

90 

95 

100 

2 

0 

0 

0.01 

0.01 

0.015 

0.02 

3 

0 

0 

0.01 

0.015 

0.022 

0.028 

5 

0 

0 

0.012 

0.015 

0.028 

0.037 

10 

0 

0.005 

0.016 

0.03 

0.035 

0.055 

Table  4:  Fraction  of  incompatible  assignment  modifications  vs.  delays 
show  that,  even  when  we  have  significant  delays  in  information  synchronization,  the  average  number 
of  tentative  assignment  modifications  (augmentations)  that  were  rejected  was  minimal,  indicating 
that  our  speculative  computation  approach  can  provide  significant  advantages  over  a  centralized 
approach. 
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3  Dynamic  Vehicle  Assignment  and  Scheduling 

In  the  previous  section,  we  summarized  our  results  on  Strategy  Development,  looking  at  algorithms 
that  compute  approaches  for  developing  task  strategies  and  allocation  without  focusing  on  detailed 
trajectory  ^control  and  scheduling.  In  order  to  generate  actual  control  trajectories,  lower  level 
algorithms  are  needed  to  compute  actual  trajectories  and  schedules  of  activities.  In  this  section, 
we  summarize  our  results  on  Dynamic  Vehicle  Assignment  and  Scheduling. 

We  consider  multiple  vehicles  headed  for  multiple  target  points  to  collect  rewards  associated  with 
them.  The  team  objective  is  to  maximize  the  total  reward  accumulated  over  a  given  time  interval. 
Complicating  factors  include  uncertainties  regarding  the  location  of  target  points  and  the  effec¬ 
tiveness  of  collecting  rewards,  heterogeneous  vehicle  capabilities  and  time-varying  rewards.  The 
problem  consists  of  assigning  tasks  and  arrival  times  for  each  vehicle,  and  selecting  trajectories 
from  each  vehicle  to  its  assigned  task.  A  motivating  paradigm  for  this  problem  is  the  assignment 
of  smart  autonomous  munitions  to  a  group  of  targets. 

The  mission  control  algorithms  discussed  in  Strategy  Development  work  with  a  highly-aggregated 
representation  of  the  continuous  trajectories  that  vehicles  must  fly.  These  aggregate  problems  serve 
to  identify  the  tasks  that  should  be  scheduled  for  individual  vehicles  by  the  lower  level  dynamic 
vehicle  assignment  and  scheduling.  The  problem  of  interest  is  controlling  the  movement  of  the 
UAVs  and  ultimately  assigning  them  to  targets  so  as  to  maximize  the  total  reward  collected  by 
visiting  targets  within  a  given  mission  time.  The  problem  is  complicated  by  several  factors:  (i) 
Target  rewards  may  be  time-dependent;  (ii)  Different  vehicles  have  different  capabilities  in  terms 
of  collecting  the  reward  associated  with  a  target;  (iii)  The  exact  location  of  targets  may  not  always 
be  known;  (iv)  There  may  be  obstacles  which  constrain  the  feasible  trajectories  of  vehicles  or  may 
cause  their  elimination  when  they  are  encountered,  (v)  Information  about  the  state  of  the  mission 
space  is  incomplete  and  dynamic. 

The  above  problems  are  complex,  and  are  often  approached  using  a  functional  decomposition  that 
solves  task  assignment  separately  from  coarse  routing  and  fine  routing.  We  developed  an  alternative 
based  on  temporal  decomposition,  solving  an  optimization  problem  with  a  receding  horizon.  An 
advantage  of  this  approach  is  that  it  integrates  the  three  tasks  of  UAV  assignment  to  targets, 
routing  of  UAVs  to  their  assigned  targets,  and  real-time  trajectory  generation,  all  in  one  function. 
The  main  idea  of  the  proposed  control  scheme  is  to  dynamically  determine  vehicle  trajectories  by 
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solving  a  sequence  of  optimization  problems  over  a  planning  horizon  and  executing  them  over  a 
shorter  action  horizon.  Thus,  we  replace  a  complex  discrete  stochastic  optimization  problem  by  a 
sequence  of  simpler  continuous  optimization  problems. 

3.1  Formulation 


In  our  research,  we  considered  a  setting  which  involves  a  team  of  N  UAVs  indexed  by  j  =  1, . . . ,  iV 
and  a  set  of  M  targets  indexed  hy  i  =  1, . . . ,  M  in  a  2-dimensional  space.  Associated  with  the  zth 
target  is  a  reward  Ri.  A  mission  is  defined  as  the  process  of  controlling  the  movement  of  the  UAVs 
and  ultimately  assigning  them  to  targets  so  as  to  maximize  the  total  reward  collected  by  visiting 
targets  within  a  given  missbn  time  T. 


In  a  2-dimensional  mission  space,  the  location  of  the  ith  target  is  denoted  by  G  i  =  1, . . . ,  AT, 
and  the  position  of  the  jth  vehicle  at  time  t  is  denoted  by  Xj{t)  G  j  =  1, . . . ,  M.  The  vehicles’ 
intial  positions  are  given  by  xjq,  j  =  1, . . .  ,M.  Assuming  a  vehicle  travels  at  constant  velocity 
throughout  the  team  mission,  the  vehicle  dynamics  are 


xj{t)=-Vj 


COS  Uj  (t) 

sin  Uj{t)  J  ’ 


=  Xjo 


where  Uj{t)  G  [0, 27r]  is  the  heading  (the  control  variable)  of  vehicle  j  and  Vj  is  the  corresponding 
velocity. 


Vehicles  complete  tasks  in  the  mission  space  by  visiting  targets.  To  distinguish  the  importance  of 
tasks  at  time  t,  each  target  has  an  associated  reward  function  denoted  by  Ri(j)i{t)^  where  Ri  is  the 
maximal  reward  and  G  [0, 1]  is  a  discounting  function  which  describes  the  changing  of  reward 
with  time.  This  allows  us  to  model  targets  that  must  be  visited  by  some  deadline  or,  more  generally, 
targets  whose  rewards  are  positive  only  within  some  given  time  window.  To  distinguish  between 
the  effectiveness  of  vehicles  relative  to  a  target  z,  we  define  a  capability  factor  Pij{t)  G  [0, 1],  which 
reflects  the  probability  that  a  UAV  j  visiting  target  i  at  time  t  will  complete  the  task  and  collect 
the  reward  Ri(j)i{t).  The  objective  of  the  mission  is  to  collect  the  maximal  total  reward  by  the  end 
of  some  mission  time  T. 
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3.2  Receding  Horizon  Control 


To  meet  this  goal,  we  developed  a  cooperative  controller  updated  at  time  points  denoted  by 
k  =  0, 1, . . during  the  mission  time.  At  tfc,  the  controller  operates  by  solving  an  optimization 
problem  Pfc,  whose  solution  is  the  control  vector  =  [ui{tk) . . .  UMi^k)]-  We  explain  next  how  Pfc 
is  formulated  (see  also  [6]). 

Suppose  that  vehicles  are  assigned  headings  wi(tfc)?  •  •  •  sit  time  tk  and  maintain  them  for  a 

planning  horizon  denoted  by  iJjt-  Then,  at  time  tk  +  Hk  the  positions  of  the  vehicles  are  given  by 


^j{ik  H"  ^j{l'k)^k 


Define 

'^iji'tk^  Ufc)  =  (tfc  +  Hk)  +  W^ji^k  +  Hk)  —  ViW/Vj  (8) 

and  note  that  Tij{tk,vik)  is  the  earliest  time  that  vehicle  j  can  reach  target  i  under  the  condition 
that  it  starts  at  tk  with  control  dictated  by  and  then  proceeds  directly  from  Xj{tk  +  Hk)  to  the 
target  point  (||  •  ||  is  the  usual  Euclidean  norm).  We  are  interested  in  the  maximal  reward  that 
vehicle  j  can  extract  from  target  i  if  it  reaches  the  target  at  time  Tij{tk^  Ujt).  Clearly,  this  is  given 
by  Ri(j>i[Tij{tk,Uk)]-  For  convenience,  define 


^ijih)  =  (l>i[rij{tk,Uk)]  (9) 

where  it  is  worth  pointing  out  that  unlike  depends  on  both  i  and  j.  It  is  also  clear 

that  the  probabilty  of  extracting  this  reward,  evaluated  at  time  tk^  is  given  by  Pij[rij{tky  ujt)].  For 
convenience,  we  set 

Pij  (4)  =  Pij  [njitk,  u*;)]  (10) 


Next,  let  us  define  the  relative  distance  function,  defined  as 


\\xj{t)-yi\\ 


(11) 


which  provides  the  proximity  of  vehicle  j  to  target  point  i  relative  to  other  vehicles,  given  all  vehicle 
positions  xi{t)^ , . .  ,XM{t)-  Based  on  this,  let  us  define  a  normalized  relative  distance  function 
QijiSij)  to  be  a  monotonically  nonincreasing  function  such  that 


QijiO)  =  1,  qijil/M)  =  1/M,  lim  %(5ij)  =  0  (12) 

Oij  » 1 
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which  we  view  as  the  probability  that  target  i  is  assigned  to  vehicle  j  at  time  t  (thus,  when  5ij  =  0 
for  example,  vehicle  j  is  ^‘committed”  to  target  i).  We  are  interested  in  the  value  of  this  function 
at  t  =  tfc  -f  and  define 

Qijif'k)  —  d"  (13) 

We  can  now  present  the  optimization  problem  P^,  formulated  at  time  as  follows: 

N  M 

““  ^ 

i=i  j=i 


with  0ij(tfc),  Pij{tk)i  and  qij{tk)  as  defined  in  (9),  (10),  and  (13)  respectively.  The  expression 
Ri^ij{'^k)Pij{tk)Qij{'^k)  111  (14)  Can  be  seen  as  the  expected  reward  that  vehicle  j  collects  at  target  f, 
evaluated  at  time  using  a  planning  horizon  Similar  to  Pij{t)^  we  can  also  define  Tij{t)  G  [0, 1] 
to  be  the  ith  target’s  capability  factor  when  inflicting  damage  to  vehicle  j.  Letting  Cj  be  the  cost 
of  losing  the  jth  vehicle,  we  can  extend  (14)  above  to  include  expected  costs  due  to  UAV  loss: 

[  N  M  N  M  1 


max 


^ i^k )Pij  (^fc ) Qij  i^k)  Cj ^ij  i^k) 

i=l  j=l  i=l  j=l 


(15) 


where  rij{tk)  is  defined  similar  to  Pij{tk)-  Note  that  this  is  a  nonlinear  optimization  problem  whose 
solution  is  generally  easy  to  obtain  compared  to  a  combinatorial  optimization  problem  seeking  to 
assign  UAVs  to  targets  over  the  entire  mission  time. 


Upon  getting  the  optimal  for  (15)  based  on  all  state  information  available  at  4,  the  UAV  team 
will  follow  this  control  for  an  action  horizon  hk<  Hk<  The  process  is  then  repeated  at  time 


^fc+l  —  d"  ^  —  0,  1,  . . . 

The  value  of  hk  is  determined  by  two  factors.  First,  if  an  unexpected  event  takes  place  at  some 
te  €  {tk,tk  d-  hk),  then  we  set  hk  =  te  ~  tk-  Otherwise,  we  simply  update  the  control  after  the 
prespecified  amount  of  time  hk-  Thus,  in  general,  {tk}  is  a  random  sequence. 

The  choices  of  the  functions  </)i(t)  and  qij{6ij)  may  depend  on  a  variety  of  factors,  including  the 
possible  presence  of  hard  deadlines  on  visiting  target  points  and  the  desirability  of  maintaining 
tight  vehicle  formations  as  opposed  to  spreading  the  vehicles  out  over  the  full  mission  space.  Some 
such  choices  were  given  in  [6]. 
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3.3  Results 


The  properties  of  the  cooperative  controller  outlined  above  were  tested  in  a  variety  of  simulated 
environments  and  found  to  match  a  reward  upper  bound  (obtained  by  an  exhaustive  search  of  all 
possible  vehicle-to-target  assignments  and  minimal  straight  line  trajectories)  with  high  probability. 
The  most  surprising  finding  of  this  work  to  date  [8],  however,  has  been  the  fact  that  vehicles 
are  always  ultimately  assigned  to  target  points^  despite  the  fact  that  our  approach,  by  its  nature, 
was  never  intended  to  actually  perform  any  such  assignment.  A  salient  property  of  this  scheme, 
which  selects  real-valued  vehicle  headings,  is  to  enforce  the  convergence  of  these  headings  to  values 
that  reflect  discrete  vehicle-to-target  point  assignments.  This  intriguing  observation  has  broad 
implications  in  tackling  such  haxd  stochastic  and  intrinsically  discrete  optimization  problems. 

Looking  at  (15),  note  that  the  solution  of  this  problem  is  a  real- valued  heading  Uj{tj^)  for  every 
vehicle  j  =  1, . . . ,  M.  There  is  no  express  constraint  imposed  to  assign  a  vehicle  to  a  target  location, 
i.e.,  there  is  no  constraint  of  the  form  Xj{tk  +  JTfc)  G  {yi, . . . ,  yjv}  or  =  ngfa+Si-Sfcill 

forcing  a  vehicle  to  either  be  at  a  target  by  a  certain  time  or  to  set  a  heading  for  it.  The  intent  of  de¬ 
signing  the  RH  controller  was  originally  to  simply  direct  vehicles  toward  points  in  the  mission  space 
in  a  way  that  the  team  maximizes  an  expected  reward  as  represented  by  the  objective  function  in 
(15);  a  lower-level  controller  might  then  be  responsible  for  making  final  discrete  assignments.  How¬ 
ever,  the  empirical  observation  made  was  that  in  fact  each  vehicle  always  eventually  ends  up  visiting 
a  target  point,  as  long  as  the  planning  horizon  is  selected  to  be  Hk  =  niinje^^ie7-{||yi  “  2^j(^fc)||  /Vj}^ 
the  smallest  distance  (in  time  units)  between  any  vehicle  and  any  target  point  at  time 

In  order  to  prove  this  convergence  property,  we  used  the  theory  of  potential  functions.  Intuitively, 
we  may  view  a  target  reward  as  the  value  of  a  potential  function  associated  with  that  target;  this 
potential  then  exerts  an  “attractive  force”  on  UAVs  that  “compete”  for  assignment  to  the  target. 
Using  this  framework,  we  introduce  the  concept  of  a  stationary  UAV  trajectory  in  the  sense  that 
it  always  ends  up  assigned  to  a  specific  target  (as  opposed  to  a  point  in  space  from  which  it  would 
be  the  responsibility  of  a  separate  controller  to  assign  it  to  a  target).  By  analyzing  the  dynamics 
of  a  simple  setting  in  [8],  we  have  obtained  convergence  proofs  for  simple  vehicle  scenarios  under 
simple  restrictions  on  the  relative  values  of  the  tasks. 

Our  discussion  above  has  been  based  on  the  assumption  that  all  target  locations  are  originally  known 
and  given  hy  yi^  i  =  . . .  ^N.  Clearly,  it  is  desirable  to  endow  a  UAV  team  with  the  versatility 
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to  adjust  its  cooperative  behavior  as  new  targets  are  detected.  The  very  nature  of  the  RH  control 
approach  makes  it  amenable  to  this  capability:  since  the  controller  is  updated  whenever  a  new  event 
occurs,  we  can  simply  define  the  detection  of  a  new  target  as  such  a  “new  event” .  Consequently, 
the  presence  of  a  new  target  causes  changes  in  (15)  when  it  is  solved  following  this  event,  but  there 
is  no  fundamental  difficulty  expected  other  than  an  increase  in  the  dimensionality  of  the  problem 
since  there  are  now  N  1  targets  rather  than  N, 

A  phenomenon  that  we  observed  in  our  work  is  that  of  occasional  oscillatory  behavior  in  the 
trajectories  of  certain  UAVs.  This  is  illustrated  in  Fig.  1  (based  on  simulation  experiments),  where 
the  RH  controller  is  used  with  two  UAVs  and  six  targets.  The  UAV  on  the  right  side  immediately 
experiences  oscillations  and  the  reason  is  that  it  happens  to  be  initially  positioned  roughly  equally 
close  to  targets  5  and  6  which  have  comparable  characteristics.  A  similar  situation  arises  for  the 
UAV  on  the  left  side  when  it  reaches  the  vicinity  of  targets  1  and  4.  Although  this  behavior  is  seen 
to  be  ultimately  resolved,  it  is  clearly  undesirable.  The  root  of  this  problem  lies  in  the  formulation 
of  (15)  which  assumes  that  a  vehicle  is  free  to  arbitrarily  change  directions  at  no  cost. 

We  eliminated  this  behavior  by  introducing  a  cost  (i.e.,  a  negative  reward  in  (15))  to  changing 
directions.  The  direction  change  cost  function  is  A(u,  u'),  where  u  is  the  current  heading  of  a 
vehicle  and  is  a  new  heading,  determined  as  the  solution  of  a  problem  of  the  form  Pfc  at  some 
tjc.  Using  this  approach,  the  oscillatory  behavior  of  of  Fig.  1  is  eliminated. 

3.4  Continuing  reseairch  direction 

There  are  several  issues  that  remain  for  future  investigation.  One  of  them  is  the  choice  of  functions 
<j>i{t)  and  QijiSij)  (and  hence  ^ij{tk)  and  qij{tk))  depends  on  several  factors.  However,  the  basic 
structure  of  these  functions  is  well-defined,  given  requirements  such  as  (12).  It  is  reasonable  to 
conjecture  that  the  effect  of  altering  the  “shape”  of  the  functions  is  not  significant  on  the  behavior 
of  the  RH  controller.  This  is  an  issue  that  deserves  further  investigation. 

Another  important  extension  is  to  deal  with  stand-off  capabilities  on  UAVs.  Often,  a  UAV  can 
perform  a  task  at  a  significant  stand-off  distance.  Platforms  such  as  Global  Hawk  have  radars 
capable  of  imaging  objects  100  km  away  from  the  platform.  An  abstraction  of  this  problem  is  to 
have  UAVs  reach  a  distance  to  a  task  in  an  interval  [dmin,  dmax]  in  order  to  be  able  to  perform  this 
task. 
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A  third  extension  is  dealing  with  search  activities.  Our  discussion  above  has  been  based  on  the 
assumption  that  all  target /task  locations  are  known  and  given  by  yi,  i  =  1,...  ,A/'.  Clearly,  it 
is  desirable  to  endow  a  UAV  team  with  the  versatility  to  search  for  new  targets.  Although  in 
principle  we  believe  the  same  approach  can  be  adopted  for  such  tasks,  this  is  a  research  direction 
that  remains  for  future  investigations. 
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4  Publications 


The  following  publications  document  in  greater  detail  the  work  discussed  above. 

1.  “A  Receding  Horizon  Approach  for  Solving  Some  Cooperative  Control  Problems,”  C.  G. 
Cassandras  and  W.  Li,  Proc.  2002  Conference  on  Decision  and  Control^  Las  Vegas,  Nevada, 
December  2002. 

2.  “Model  Predictive  Control  for  Dynamic  Unreliable  Resource  Allocation,”  D.  A.  Castahon 
and  J.  M.  Wohletz,  Proc.  2002  Conference  on  Decision  and  Control,  Las  Vegas,  Nevada, 
December  2002. 

3.  “Model  Predictive  Control  for  Stochastic  Dynamic  Resource  Allocation,”  D.  A.  Castahon, 
submitted  to  IEEE  Transactions  on  Automatic  Control. 

4.  “Distributed  Algorithms  for  Reassignment,”  D.  A.  Castahon  and  C.  Wu,  Proc.  2003  Confer¬ 
ence  on  Decision  and  Control,  Maui,  HI,  December  2003. 

5.  “  Stability  Properties  of  a  Cooperative  Receding  Horizon  Controller,”  C.  G.  Cassandras  and 
L.  Wei,  Proc.  2003  Conference  on  Decision  and  Control,  Maui,  HI,  December  2003. 

6.  “Stability  Properties  of  a  Receding  Horizon  Contoller  for  Cooperating  UAVs,”  W.  Li  and  C. 
G.  Cassandras,  to  appear  in  Proc.  of  43rd  IEEE  Conf.  Decision  and  Control,  2004. 

7.  “A  Cooperative  Receding  Horizon  Controller  for  Multi- Vehicle  Uncertain  Environments,”  Li, 
W.,  and  Cassandras,  C.G.,  subm.  to  IEEE  Trans,  on  Automatic  Control,  2004. 

8.  “Planning  and  Control  of  UAVs  as  a  Dynamic  and  Stochastic  System  ,”  D.  Ahner  and  D.  A. 
Castahon,  presented  at  MORS  2004,  Monterey,  CA. 
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